Reliability of a seminar grading rubric in a grand rounds course

PURPOSE Formal presentations are a common requirement for students in health professional programs, and evaluations are often viewed as subjective. To date, literature describing the reliability or validity of seminar grading rubrics is lacking. The objectives of this study were to characterize inter-rater agreement and internal consistency of a grading rubric used in a grand rounds seminar course. METHODS Retrospective study of 252 student presentations given from fall 2007 to fall 2008. Data including student and faculty demographics, overall content score, overall communication scores, subcomponents of content and communication, and total presentation scores were collected. Statistical analyses were performed using SPSS, 16.0. RESULTS The rubric demonstrated internal consistency (Cronbach's alpha = 0.826). Mean grade difference between faculty graders was 4.54 percentage points (SD = 3.614), with ≤ 10-point difference for 92.5% of faculty evaluations. Student self evaluations correlated with faculty scores for content, communication, and overall presentation (r = 0.513, r = 0.455, and r = 0.539; P < 0.001 for all respectively). When comparing mean faculty scores to student's self-evaluations between quintiles, students with lower faculty evaluations overestimated their performance, and those with high faculty evaluations underestimated their performance (P < 0.001). CONCLUSION The seminar evaluation rubric demonstrated inter-rater agreement and internal consistency.